Machine Learning in Medicine
Digital clinical data, captured and stored on electronic medical records by hospitals and clinics, along with ever growing volumes of medical imaging data, have sparked growing interest and applications of machine learning in medicine.
In recent years there has been a proliferation of artificial intelligence (AI) tools and resources available in medicine, especially with ever increasing computing power and a growing acceptance of cloud computing by hospitals and clinicians. Imaging analysis and clinical decision support are two particularly popular applications of machine learning in medicine, with tools that support diagnosis, treatment, care coordination and remote monitoring.
There is much promise in the utilization of AI methodologies such as machine learning and deep learning for augmented biomedical image interpretation in radiology, cardiology, pathology, dermatology, ophthalmology and genomic medicine.
One example of machine learning for medical imaging involves differential diagnosis of breast cancer enabled by joint analysis of functional genomic information and pathology images (pathogenomics) within a biomedical imaging informatics framework consisting of image extraction, feature combination, and classification.
Algorithms based on deep convolutional neural networks have been used to detect diabetic retinopathy in retinal fundus photographs with high specificity and sensitivity, as good as with board-certified ophthalmologists in making diagnoses.
Personalized precision medicine with all its complexity and enormity of data to be analyzed is particularly well suited for the portfolio of AI methodologies, including deep learning, which can be used to identify and assess patients with similar genotype-phenotype characteristics. In genomic diagnostics, clinicians are often frustrated by the tedious nature of searching for genotype-phenotype interrelationships among syndromes, especially for extremely rare diseases. Now, geneticists may be able to use visual diagnostic decision support systems that employ machine learning algorithms and digital imaging processing techniques in a hybrid approach for automated detection and diagnosis in medical genetics.
An essential part of the precision medicine paradigm is individualized therapy based on genotype-phenotype coupling and pharmacogenomic profiles. There are many potential applications of deep learning for large datasets in pharmaceutical research, such as physicochemical property prediction, formulation prediction, and properties such as absorption, distribution, metabolism, excretion, toxicity, and even target prediction.
Surgical robotics have advanced to include 3D visualization and informatics-enriched imagery guided interventions.
Machine learning algorithms can also be applied to large-scale wearable sensor data in neurological disorders such as Parkinson's disease to significantly improve clinical diagnosis and management. Sensor-based, quantitative and objective systems for assessing Parkinson's disease have the potential to replace traditional qualitative and subjective ratings by human interpretation.
An essential part of digital medicine and wearable devices is the data mining of the incoming data for anomaly detection, prediction, diagnosis and clinical decision making. Data mining processes for data streams from wearable devices typically include feature extraction/selection processes to improve detection, prediction, and decision making by clinicians.
Machine learning techniques include supervised methodologies such as neural networks, support vector machines, naïve Bayesian classifiers, and hidden Markov models, as well as semi-supervised methods that can be used with less labeled data. These techniques can be applied to molecular imaging modalities with promising application for clinical diagnosis.
Four types of machine learning deep learning, reinforcement learning, transfer learning and one-shot learning - may figure prominently in future applications of AI in medicine.
Deep learning with all its myriad capabilities may well be used for many applications in medical data analytics. The multiple layers of neural nets can be assigned to the many phenotypic as well as genomic expressions of conditions such as clinical measurements, biomarkers, imaging data, genomic information and disease subtypes.
Reinforcement learning is ideally designed for the many decision making aspects of medicine since it readily accommodates recognition of complex patterns, long-term planning, and many decision-making processes in clinical practice.
Transfer learning occurs when a network that is trained for one task is then used to configure the network for another task.
One-shot learning can bring a special dimension to unique cases in medicine as it does not require the usual large dimensionality of data that the other types of machine learning techniques typically require.
Natural language processing [NLP] includes machine learning techniques for speech recognition and identification, as well as language understanding and generation. Medical NLP may become increasingly useful for collaborative curation, annotation and tagging of medical imagery data by heterogeneous teams of medical minds and machines. Curated medical images, annotated and tagged as medical “ground truth”, will become increasingly important not only for clinical detection, diagnosis and decision support, but also for the training, testing and validation at scale of machine learning algorithms requiring voluminous imagery data sets.
Personalized precision medicine may require disruptive computational platforms for new biomedical knowledge discovery, and scalable computational frameworks that can leverage hypergraph-based data models and query languages that may be well-suited for representing complex multi-lateral, multi-scalar, and multi-dimensional relationships. Hypergraph-like stores of clinical information (e.g., from disease registries) can be combined with an individual patient's genomic and other phenotypic information (such as imaging data) to create more precise and personalized genome-based knowledge stores for clinical translation and discovery. Patients of very similar genomic and clinical elements could then be better discovered and matched for diagnostic and therapeutic strategies.
Cloud computing and storage can facilitate a full range of AI techniques for multi-institutional collaborations that may become essential to driving future applications of AI in biomedicine and healthcare. The internet of medical things (IoMT) may also provide the critical data sources for medicine in the form of wearable and monitoring devices from both hospital and home.
Clinical data analytics will increasingly rely on machine learning tools and techniques to answer many clinical questions for intelligence-based medicine, rather than current best practices of principally relying upon published medical reports for evidence-based medicine.
There is a compelling need for informatics-enriched innovation with AI-powered technologies that can improve diagnostics and therapeutics, and help deliver value-based care. The convergence of “big data” stores, improved AI algorithms, increasing use of graphical processing computational power (GPU), and cloud storage has begun to produce some intriguing machine learning projects with promising results for biomedicine and healthcare. Perhaps more importantly, continuing advances with AI-powered tools and techniques in healthcare will require efforts to ensure more collaborative teamwork and better sharing of curated datasets among the various stakeholders.
Productive AI strategies may involve synergistic collaborations of humans and machines clinicians and data scientists, empowered with AI—so that machine learning in medicine may become a key enabler of new clinical knowledge and augmented medical intelligence for learning health care systems.
Collaborative Clinical Workflows with Enterprise Imaging
The HIMSS-SIIM Collaborative Workgroup has defined Enterprise Imaging as:                “The management of all clinically relevant content, including imaging and multimedia, for the purposes of enhancing the electronic health record through a set of strategies and initiatives designed and implemented across the healthcare enterprise. These strategies and initiatives are based on departmental and specialty workflows for all clinical imaging content, and include methods for capture, indexing, management, storage, access for retrieval, viewing, exchange and analytics.”        
Enterprise imaging (EI) platforms typically provides the infrastructure, modalities, devices, and integration points, as well as a standards-based repository for storage of both DICOM and non-DICOM clinical images and video. Those centralized image repositories e.g., a vendor neutral archive or an enterprise wide PACS system typically include indices of both image and metadata-information contents held in the archive.
Medical imaging archives are increasingly becoming modality agnostic, modality vendor agnostic, specialty and service line agnostic, and viewer agnostic. Standards-based interfaces and communications, including DICOM, HL7, and standards-based Web Services, connect, enable, and support image acquisition workflows across modalities and departments. Image acquisition devices that support these standards may store their images, with meta-information, into the VNA. Acquisition devices that are supported include departmental DICOM imaging modalities, point-of-care acquisition modalities, handheld device photo or video apps, digital capture systems in procedure rooms, image exchange gateways, and software designed to import content saved on a disk or received by referring or patient portals.
Clinical content and multimedia content span four broad categories of medical workflows within Enterprise Imaging: diagnostic imaging, procedural imaging, evidence imaging, and image-based clinical reports.
Medical workflows across many departments capture and create a variety of types of “multimedia” information that is important to preserve, correlate with the images, and make accessible via the patient medical record. Multimedia content includes waveforms, audio or video clips, as well as other forms of graphical content that summarize imaging results with the results from other medical procedures and tests. Non-radiological examples can be found in many specialties including Cardiology, Neurology, Gastroenterology, Ophthalmology and Obstetrics. Graphical “report-style” results from various medical departments are increasingly being created and saved as PDF objects. These can include embedded images that show key findings, graphical diagrams that show the area of interest, or other measurement or test result information that correlates with the images.
Other examples of related multimedia content include time-based waveforms such as those produced by ECG or EEG devices. These may be treated as documents or image-like objects. Waveforms may be recorded and stored in a raw or processed form that requires an application to display them, or in some human-readable rendered form (like a PDF or screenshot). Like images, waveforms too can be classified as both evidence and diagnostic. Waveforms are the graphical representation of discrete data points but may be used as the sole basis of interpretation when other tools for analysis of discrete data points are not available or routinely incorporated within the interpretation protocol.
Most types of multimedia content, including waveforms, PDF reports, MPEG video clips, and JPEG photos, can be DICOM wrapped and stored as DICOM objects or they can be treated as a native document type (e.g., PDF, JPEG, MPEG, etc.) and saved in systems that can manage them as native objects. An important consideration is how this information will be managed, correlated, accessed, and viewed by physicians and patients. Wherever possible, related patient images and multimedia content could be made readily discoverable and shown together in a useful, natural way.
DICOM provides support for encoding both generic identification and modality and specialty-specific acquisition context for all enterprise imaging modalities. DICOM-like metadata can also be added to other image file formats like JPEG or TIFF. Other alternatives include encapsulating the image in a different standard format, such as HL7 Clinical Document Architecture (CDA), as is defined by the IHE Scanned Document (XDS-SD) profile, so that metadata remains directly associated with their related medical images.
The invention described herein supports both approaches to encapsulating and saving medical metadata together with their associated medical imagery.
Video Collaboration with Medical Imaging
This invention relates to a videoconferencing system for ‘live’, i.e., real time, near real time or minimally latent, viewing of streaming medical imagery, and more particularly, to a network system and methods of using said videoconferencing system with both medical and non-medical imagery, and multiple input operators (participant “cognitive collaborants”), each viewing the other's inputs collaboratively and concurrently.
In the past, video conferencing systems could be summarized as enabling a plurality of users systems connected to each other, each being adapted to display a work area on a display screen or connected through a computer network. Collaboration of work is done on each system by use of a management table for registered node identification codes given for each system user. That is, every computer system, or one system, requires storage of collaboration user identifier in at least one of the user's computer system. The novelty of the current invention—a system and methods of multimodal cognitive communications, collaboration, consultation and instruction for use with medical imagery - has improved upon prior art by allowing modular and scalable network clusters of gateway streamer servers that enable dynamic control allowing for faster and more efficient performance, as well as enabling for multiparty cognitive collaboration with medical imagery in a Digital Imaging and Communications in Medicine environment, hereinafter referred to as DICOM.
The DICOM Standard pertains to the field of medical imaging informatics. The DICOM Standard is well known in the arts and facilitates interoperability of medical imaging equipment by specifying a set of protocols to be followed by devices claiming conformance to the standard. The DICOM Standard outlines syntax and semantic of commands and associated information which can be exchanged using these protocols. For media communication, it provides a set of media storage services to be followed by devices claiming conformance to the DICOM Standard, as well as a file format and medical dictionary structure to facilitate access to the images and related information stored on interchange media. DICOM data file format is data formatted in groups of information, known as Data Sets. The DICOM Standard provides a means to encapsulate in a single file format structure the Data Set related to a DICOM information object. The DICOM Standard requires a single file format structure, as the DICOM Standard specifies that each DICOM file contain both File Meta Information and a properly formatted Data Set (as specified in DICOM Standard 3.10). The DICOM Standard further specifies that the byte stream of the DICOM Data Set be placed into the file after the DICOM File Meta Information (as specified in PS 3.10 DICOM Part10: Media Storage and File format for Media Interchange).
The DICOM Standard specifies the rules for encapsulating DICOM Data Sets in the requisite DICOM File format. The DICOM Standard requires that a file meta information header be present in every DICOM file, and that the file meta information includes identifying information of the Data Set (PS 3.7-1). The DICOM Standard requires that the Data Set conform to the service-object pair (SOP) Class specified in the file meta information. “The DICOM File format provides a means to encapsulate a File the Data Set representing a SOP Instance relating to a DICOM Information Object.” The DICOM Standard provides for the encapsulation of waveform data (PS 3.5 Part 5: Data Structures and Encoding), and for the encapsulation of structured reports (Supplement 114: DICOM Encapsulation of Clinical Document Architecture Documents) within imagery bit streams to facilitate the interchange of information between digital imaging computer systems in medical environments.
The DICOM File Meta Information includes identifying information on the encapsulated DICOM Data Set. The DICOM Standard requires that a file header of identifying information be present in every DICOM file. The DICOM file header consisting of a 128 byte File preamble, followed by a 4 byte DICOM prefix, followed by the File Meta Elements. This means, for example, that a DICOM file of a chest x-ray image actually contains the patient identification within the file, so that the image can never be separated from patient information by mistake. A DICOM file contains both the image and a large amount of patient information about whom, where, and how the image was acquired, known in the arts as patient metadata.
However, DICOM files often contain little information about the content of the imagery or meaning of the imagery pixels, the encapsulated waveform data used for audio clinical notes, or the encapsulated structured reports used for clinical documents, all of which are used for clinical detection, diagnosis and treatment of disease. This network system improves upon and applies in a collaborative environment which provides for capture, retrieval and concurrent viewing of both live and archived medical imagery streams for communication, collaboration and consultation with one or more sources of streaming imagery data by one or more users, also known as participant cognitive collaborants. Collaborated medical imagery streams comprise one or more sources of streaming imagery data, including DICOM imagery files. As used herein, DICOM imagery files include modality information objects, (e.g. streaming video), waveform information objects (e.g. voice audio, echocardiogram), and structured report document information objects (e.g. clinical documents), as specified in PS 3.3 Part 3: Information Object Definitions of the DICOM Standard.
Medical imagery streams include DICOM imagery files. This network system allows for each user to collaborate simultaneously with all users viewing every other users' work product, as the work product is being created, all coincident with one or more streams of streaming imagery data wherein each server manages streams of medical imagery together with participant cognitive collaborant input illustrations for use with DICOM imagery files. The network system provides live video and audio communication, as well as a method of viewing, recording and transmitting streaming imagery data, which include DICOM imagery files, in DICOM format, which requires a single file format structure. Streaming imagery data includes both live and archived imagery data. As used herein, multi-channel streaming imagery data is defined as a collection of one or more sources of streaming imagery data each of which comprise at least one image frame that defines a time progression of output from various sources, which include video, encapsulated waveform data, and encapsulated structured reports.
The network system provides multi-channel multiplexed capability for capture, retrieval and concurrent viewing of both live and archived medical imagery streams for communications, collaboration, consultation and instruction with one or more sources of streaming imagery data by participant cognitive collaborants. Participant cognitive collaborant input illustrations as defined herein include, but are not limited to telestrations, drawings, sketches, text annotations, including letter character text and numeric character text, image annotations, wave form annotations, voice annotations, video annotations, augmented reality imagery annotations, 3D/4D imagery annotations, outcomes annotations, costs annotations, resource consumption/utilization annotations, haptic annotations, patient metadata, imagery metadata, semantic metadata and annotations, appended patient metadata, appended imagery metadata and appended semantic metadata and annotations. The network system appends participant cognitive collaborant input illustrations to streaming imagery data, and encapsulates and saves those input illustrations, together with streaming imagery data, and relevant imagery metadata and semantic metadata and annotations, including appended imagery metadata and appended semantic metadata and annotations, from the collaboration session in single file format structures, known as collaborated imagery files. The ‘single file encapsulate and save’ functionality of the network system encapsulates and saves collaborated imagery files in single file format structures, as may be required or allowed by standards for clinical documentation or medical records storage, including those as specified in the DICOM Standard (e.g. as DICOM files).
The network system appends metadata tags to participant cognitive collaborant input illustrations and encapsulates and saves those tagged input illustrations together with the Data Set from the streaming imagery data and relevant metadata information from the metadata header in single file format structures for use within a DICOM imagery environment, including those as specified in the DICOM Standard. The network system appends metadata tags to alpha-numeric text annotations, image annotations, wave form annotations, voice annotations, video annotations, augmented reality imagery annotations, 3D/4D imagery annotations, outcomes annotations, costs annotations, resource consumption/utilization annotations, haptic annotations and clinical documents and encapsulates those alpha-numeric text annotations, image annotations, wave form annotations, voice annotations, video annotations, augmented reality imagery annotations, 3D/4D imagery annotations, outcomes annotations, costs annotations, resource consumption/utilization annotations, haptic annotations and clinical documents and saves those as DICOM files. The network system can also append annotation files encapsulated as DICOM files to the Data Set for streaming imagery data, and encapsulate them together with relevant metadata information from the metadata header for streaming imagery data, and save in single file format structures as collaborated imagery files (CIF).
Collaborated imagery files, also known as CIFs, conform to the DICOM Standard and can be stored, archived, queried, and retrieved as DICOM files. CIFs can be stored locally in media libraries and later retrieved for subsequent use in collaboration sessions. CIFs conform to the DICOM Standard [3.10] and can be encrypted and/or transmitted over networks for remote viewing, communication and collaboration. CIFs conform to specifications of the DICOM Standard for secure encapsulation of DICOM objects in a clinical document architecture (CDA). As such CIFs can be stored as in archives conforming to health level seven (HL7), integrating the healthcare enterprise (IHE), cross-enterprise document sharing (XDS), cross-enterprise document sharing for imaging (XDS-I), Extensible Markup Language (XML), in Tagged Image file format (TIFF), as well as in RDF triples and RDF/XML for metadata model specification.
CIF's can also contain encapsulated and saved haptic imagery and annotations in COLLADA-compliant dae files. COLLADA (collaborative design activity) is an interchange file format for interactive 3D applications that has been adopted by ISO as a publicly available specification, ISO/PAS 17506. COLLADA defines an open standard XML schema for exchanging digital assets among various graphics software applications that might otherwise store their assets in incompatible file formats. COLLADA documents that describe digital assets are XML files, usually identified with a .dae (digital asset exchange) filename extension.
CIFs conform to specifications of the DICOM Standard for encapsulation of audio with imagery data sets. CIFs conform to specifications to the DICOM Standard for DICOM structured reporting. CIFs can be viewed as stand-alone medical imagery, or embedded into other CIFs as video, audio and haptic annotations. The network system can create collaborated imagery studies, also known as CIS's, which include one or more collaborated imagery files, encapsulated and saved in single file format structures, as may be required or allowed by standards for clinical documentation or medical records storage, including those as specified in the DICOM Standard format. Collaborated Imagery Studies, also known as ‘Clini-DOCx’ are visual story boards can be used for capture, display, file exchange, publication and distribution of collections of clinical cognitive vismemes.
The DICOM Standard defines the characteristics of a medical study performed on a patient as, “a collection of one or more series of medical images, presentation states, SR documents, overlays and/or curves that are logically related for the purpose of diagnosing a patient. Each study is associated with exactly one patient” (PS 3.3 A.1.2.2 STUDY IE). Streaming imagery data can include both collaborated imagery files and collaborated imagery studies. Both CIFs and Clini-DOCx can be incorporated into medical image streams of live or archived streaming imagery data for use during synchronous or asynchronous collaboration sessions.
The traditional way of capturing an image from a medical imaging device commonly called a modality, generally consisted of an operator or technician first conducting a scan. Then, using the modality to save the image, in still or motion video format, into the modality memory or into a main image storage database. The next step in the process typically involved downloading the image into a hospital database, known in the arts as a Picture Archiving and Communications System, hereinafter referred to as PACS or PACS server. PACS is a medical imaging technology which provides economical storage of, and convenient access to, images from multiple modalities (source machine types). Electronic images, including patient information known in the arts as patient metadata, are transmitted digitally to and from PACS, eliminating the need to manually file, retrieve or transport film jackets. The universal form of PACS image file storage and transfer is the DICOM Standard, and is well known in the arts. PACS can be further defined by a storage and management system for medical images.
In the medical field, images such as x-rays, MRI's and CAT scans typically require a greater amount of storage than other images in other industries. A clinician would access the PACS system to retrieve the image, view and review the image, and conceivably develop a diagnosis based on the information from the image. This system imagery is viewed by a user and diagnosis made without image delay and the user accomplishes all these tasks live. “Live” referring to events simulated by a computer at the same speed that they would normally occur in real life. In graphics animation, for example, a live program (such as this inventor's system) would display objects moving across the display at the same time they would actually move, or in the case of this invention, a cognitive collaborant views the image live and collaborates from cognitive collaborant to cognitive collaborant with no perceivable delay to any of them.
The inventor has developed a novel and simple network system and methods of using the same, to allow a group of cognitive collaborants to concurrently collaborate on a computer system, with each participant cognitive collaborant viewing each other's telestrations, drawings, and annotations and saving them together with streaming imagery data, annotations and relevant imagery metadata, including appended imagery metadata and semantic metadata and annotations, and saving them together in single file format structures as may be required or allowed by standards for clinical documentation or biomedical records storage, including those as specified in DICOM, C-CDA and FHIR Standards for interoperable health information exchange.